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ABSTRACT 



This paper describes a study designed to investigate the 
effectiveness of narrative versus step-by-step instructions for a computer 
task. The participants in this study were 31 undergraduate education students 
enrolled in a computer literacy class at the University of Memphis during the 
Summer 1996 semester; none of the participants had prior knowledge of e-mail. 
Participants were randomly assigned to one of the two treatment groups and 
were individually observed as they completed the steps for retrieving, 
replying, spell checking, and sending e-mail using software on a mainframe 
computer. After completion of their respective treatments, participants 
completed a survey designed to assess their attitudes toward the instruction. 
There was no performance time difference between the two groups, but the 
step-by-step treatment made fewer errors during the more complex tasks. There 
was a lack of difference in attitude between the two groups. A figure 
illustrating the performance rating form used by the researchers is included, 
as well as tables containing data on errors on task and errors by treatment. 
(DLS) 
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Abstract 

The purpose of this study was to investigate the effectiveness of narrative versus step-by-step instructions for a 
computer task Naive participants were observed as they completed the steps for retrieving, replying, spell checking, and 
sending email using software on a mainframe computer. There was no performance time difference between the two 
groups, but the step-by-step treatment made fewer errors during the more complex tasks. 

One problem often slighted in the software development process is the development of appropriate 
documentation for the user. This observation is supported by the growing number of after-market software books and the 
growing need for software technical support. Although there are many examples of excellent software manuals, there are 
also numerous examples of manuals that offer little more than a programmer’s description of how the software works. 
An analysis by Kaplan (1989) revealed five types of software documentation: reference manual, tutorial manual, user (or 
combination) manual, quick reference card, and on-line help. She believed that for novices, . . the tutorial is the most 
essential piece of documentation” (p.32). One problem with tutorials is the writer’s potential for underestimating the 
intelligence of the user and sinking to the “Dick and Jane” level (Deterline, 1988). 

McGehee (1984) defined tutorials as “a cookbook, hand-holding, step-by-step method of teaching the concepts 
and techniques of using a program” (p.57). It is interesting to note that McGehee’ s definition uses the term step-by-step, 
which suggests a non-narrative approach. Tutorials are recommended by McGehee if the manual is the user's first 
introduction to a program, if the program is complex, and if the user is a computer novice. The major problem identified 
with writing tutorials is trying to match the user's knowledge, a problem that depends on the particular software and the 
intended audience. Deterline (1988) issued a plea for inclusion of user needs when documentation is written, because 
much software documentation is written by subject-matter experts (SME), often the computer programmers and it is very 
difficult for non-experts to comprehend. 

If the documentation manuals are the user's first contact with the software, how should the information be 
presented? Brockmann (1990) discussed the limitations of tutorials in general, which he believes may constrain users to 
a set of minimum expectations. Tutorial material is “usually organised around user tasks or around a hierarchy of user 
needs” (p. 74) In contrast, Thurston (1986) states that general purpose computer programs (e.g. programming 
languages) are better served with narrative form tutorials, while narrowly defined programs (e.g. accounting packages) 
are best dealt with using step-by-step tutorials. Webb (1989) also recommended step-by-step tutorials for procedures and 
narrative form for overviews or introductions. “Cookbook” tutorials which consist of simple action sentences (e.g. Turn 
on computer. Click on icon for program. Select word processing) differ in form from numbered steps which move users 
sequentially through a procedure tutorial (e.g., 1. Turn on computer. 2. Locate the program’s icon. 3. Double click on the 
icon. 4. When the program starts, find the box for word processing and click on it). Tutorials, regardless of form should 
begin with a general overview and include as many illustrations as possible. However, Kaplan (1989) suggests that 
narrative style instruction often hides important messages in a mass of text. Although the use of numbered steps makes 
logical sense for complex tasks, a review of the instructional design literature failed to identify any research comparing 
narrative versus step-by-step approaches to teaching procedures. 

The purpose of this study was to investigate the effectiveness a narrative versus step-by-step presentation of a 
procedure for naive computer users. It was predicted that the step-by-step approach would be more time efficient, since a 
narrowly focused application (email) was used in the study. It was also predicted that participants receiving the step 
treatment would make fewer overall errors and fewer errors of each type. We were also interested in knowing how the 
two treatments influenced participants’ attitudes toward the instructional materials. 
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Method 

Participants 

The participants in this study were 3 1 undergraduate education students enrolled in a computer literacy course 
at the University of Memphis during the Summer, 1996 semester. The participants generally had little computer 
experience and many suffered from “computerphobia.’* None of the participants had any prior knowledge of email. 
Participants were randomly assigned to one of the two treatment groups. 

Materials 

The treatment material consisted of a three part instructional unit providing information on how to log onto the 
University EMAIL system, read and compose email, spell check, reply to, and send messages. The instructions 
encouraged students to proceed at their own pace. 

Lesson Description 

A unit of instruction was on how to use the University’s VAX EMAIL server, compose, read, spell check, reply 
to, and send EMAIL messages using the VAX software. The first treatment group was the narrative treatment (n=15). 
This treatment group read text presented in a traditional narrative format with each paragraph including from three to 
seven steps. The following is a sample of the narrative treatment. 

Turn on your computer. After the computer has finished its start-up process, you will see the “Finder” screen. 
Use the mouse to move the arrow to the Apple icon (picture) in the top left corner of the screen. With the arrow 
pointing at the Apple icon, press the mouse button and hold it down. 

The second treatment group (n=l 6), step-by-step, read instructions that were clearly enumerated and separated 
from each step. The following is a sample of the step-by-step treatment. 

1. Turn on your computer. 

2. After the computer has finished its start-up process, you will see the “Finder” screen. 

3. Use the mouse to move the arrow to the Apple icon (picture) in the top left corner of the screen. 

4. With the arrow pointing at the Apple icon, press the mouse button and hold it down. 

The two treatments contained the same information but varied in the way it was displayed on the page. The only 
difference was the numbering of the steps in the second treatment. 

Instruments 

After completion of their respective treatments, participants were given a survey designed to assess their 
attitudes toward the instruction. The attitude survey consisted of six core items that asked participants to rate the lesson, 
computer learning, the VAX system, the usefulness of email, this lesson, the sufficiency of items, and future email use 
using a five point Likert scale (l=strongly disagree, 5=strongly agree). The second instrument was an observation form 
used by the experimenter during the treatment The form listed each step of the process with places to indicate if the step 
was performed correctly and in sequence, performed incorrectly, performed correctly out of sequence or skipped. There 
was also a place for noting the starting and ending times and a section for noting participant behaviors (see Figured). 
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.Figure I. Performance Rating Form 
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Procedure 

Participants were scheduled to complete the instruction in a one-on-one setting with the experimenter. Each 
was seated at a color Macintosh computer (LCn or LCIQ) and given the printed instructions. The starting and ending 
times were noted for each participant. The experimenter observed each subject complete the tasks. The sequence of steps 
completed and the type of errors made were recorded by the experimenter using the following instrument. Each time a 
participant attempted a step, the participant’s action was evaluated and recorded on the checklist. 



Results and Discussion 

The first analysis was a comparison of completion times and overall errors for each of the treatments with T- 
tests and using SPSS statistical software. No significant differences were found for time on task (times ranged from 19 
to 23 minutes), or overall errors by treatment (overall errors ranged from 0 to 4). 

The errors in the 13 steps in the procedure were summed into four groups: Steps 1-4 (initial logon); Steps 5-7 
(composing and sending message); Steps 8-10 (logon and reading message); and Steps 11-13 (replying, spell checking 
and sending message). Only the final group of steps varied significandy by treatment when compared using MANOVA, 
with the step procedure group making fewer errors (F=5.0024). 

Table 1. Errors on Task 



Variable 


Treatment #1 (Narrative) 


Treatment #2 (Step) 


F. Prob. 




Mean 


S. D. 


Mean 


S.D. 


p<.05 


Error Steps 1-4 


.200 


.414 


.563 


.814 


.1329 


Error Steps 5-7 


.600 


.632 


.313 


.479 


.1626 


Error Steps 8-10 


.067 


.258 


.000 


.000 


.3097 


Error Steps 11-13 


.533 


.640 


.125 


.342 


.0332 



When type of errors by treatment were compared with MANOVA, only the Wrong Command Key errors 
differed, with the step procedure treatment group making significantly fewer errors (F=7.1382). 



. Table 2. Errors by Treatment 



Variable 


Treatment #1 (Narrative) 
Mean S.D. 


Treatment #2 (Step) 
Mean S.D. 


F. Prob. 

p<.05 


Wrong Command Key 


.733 


.704 


.188 


.403 


.0122 


Wrong Entry Typed 


.400 


.632 


.750 


1.065 


.2790 


Wrong Sequence 
Worked 


but .133 


.352 


.063 


.250 


.5212 


Wrong Sequence Error 


.200 


.414 


.000 


.000 


.0629 



MANOVA comparisons found no other significant differences between groups for Looks at Instructions, Asks 
for Help, or Shows Frustration. The attitude survey questions also revealed no significant differences between 
treatments, although both treatment groups recorded positive overall attitudes toward the instruction. 

The lack of significant differences in times for the treatment groups may have resulted from a motivated 
participant group. All participants were required to learn to use email in their ongoing literacy course which may have 
resulted in a greater degree of learner engagement than might have otherwise occurred. Thus there was no support for the 
first hypothesis (the step approach will be more time efficient). This finding is in agreement with a study by Morrison, 
Ross, Case and G opal akrish nan (1995) which found that purposeful engagement raised scores. 

Similarly, there were no significant differences in overall errors by treatment in this study. This lack of 
difference may be due to the relative simplicity of the procedure, with few opportunities for errors. However, there were 
significant differences in performance on the most difficult part of the procedure (replying to a message, spell checking 
and sending the reply). The step-by-step group made fewer keystroke errors. This finding suggests that step-by-step 
procedures may produce fewer errors when the procedure is more complex. Participants using the narrative approach 
found this group of steps very difficult and often showed frustration while attempting to complete the tasks. The step-by- 
step group appeared to have gained enough confidence in their ability that they were not as intimidated by this portion of 
the lesson. 

Significant differences were also found for Wrong Command Sequence errors. A Wrong Command Sequence 
error occurs when a participant presses the wrong key and is not successful at completion of the step. This result 
appeared to be due to a loss of concentration by participants receiving the narrative treatment. These participants often 
lost their place in the procedure because they lacked the number cues provided to the step-by-step participants. This 
finding is consistent with Thurston’s (1986) recommendation that step-by-step procedures are more appropriate for 
narrowly defined applications such as email. 

The lack of differences in attitudes by treatment may be explained by the participants’ need to acquire the skill 
taught by the lesson. Some negative behaviors were observed during the treatments (mostly impatience over having to 
read the instructions), but the majority of the participants were positive (and often enthusiastic) about their new skill. 
Another factor that may have influenced performance and attitudes was the individualized testing situation. This 
approach was in direct contrast to the 20:1 student-instructor ratio in the computer literacy class. This one-on-one 
environment may have increased their confidence. Also, the treatment instructions were considerably more detailed than 
those provided in the textbook. 

Most of the students (n=28) were forced to seek help from the experimenter at least once because of confusion 
over the procedure, especially when asked to choose a new password (problems included new password already in use 
and illegal password selected). Almost half (n=13) showed signs of frustration, especially when trying to press the 
CONTROL-Z key combination, which is a unique keystroke on a Macintosh. No participants attempted to explore the 
menus or other options and only one participant tried to send any additional messages. 

It appeared from the observations that the major determining factor of success was the participant’s motivation, 
not the format of the written procedure. Participants who experienced difficulty appeared to skip portions of the printed 
lesson and skimmed longer explanations which resulted in less understanding and more errors. Participants with higher 
motivation completed the tasks with few mistakes and displayed interest in the underlying structure of email. Those with 
lower motivation often ignored instructions (especially concerning spell checking) and showed little overall interest 

The differences in Control Key sequence errors indicated that the^step procedure produced fewer errors. One 
possible explanation is that the more complex the task and the more critical the order of the steps, the more advantageous 
are the step procedures. 



Overall, the results were less conclusive than predicted, but significant differences in two of the variables 
examined (Steps 11-13 and Wrong Command Sequence) indicates that there are measurable differences between the two 
methods of instruction. 

Recommendations 

The study indicated that differences in effectiveness of step-by-step procedures may be less than generally 
believed. The results of the study suggests additional research with complex procedures should be used to determine if 
complexity of task plays a part in differentiating methods of instruction. Future studies should include a variety of tasks 
that are simple and complex, short and long, and familiar and unfamiliar to the participant. 
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